Do you have one or more favorite websites you would like to search using DEVONagent Pro? While the potential to create a bespoke plugin depends on the specific site, DEVONagent Pro has a built-in tool to aid in creating a site-specific plugin. This chapter guides you in using its integrated plugin assistant to create an XML plugin, walking you through a real-world, practical example.
Now let's create a simple functional plugin from scratch!
Step 1: Open a browser window for the Wiby.me search engine in DEVONagent Pro or any other web browser. Search for awk.
At the bottom of the results, look for the control to jump to the next page of results, e.g., "Next", a list of numbers, or in this case "Find More". In DEVONagent Pro you should see the link in the address bar at the top of the window. Control-click the text and copy the link. Leave this window open while developing the plugin.
Step 2: Choose File > New Plugin. This will open the Plugins & Scanners window. In the URL field, paste the copied link, and replace awk in the URL with _agentQuery_. The _agentQuery_ placeholder is replaced with your search terms when using the plugin. Often there is a code in the URL directing how the site navigates through result pages, e.g., start= or next=. In this case it's p= and take note it is 2. We'll use this to make an adjustment after saving the plugin. Replace the number with the _agentOffset_ placeholder, used by DEVONagent Pro to jump from result page to result page. Enter a name for the plugin and optionally, a description. The Search Engine radio button is automatically selected by DEVONagent Pro. Then press the Add button.
DEVONagent Pro assumes the start page is zero-based and many sites are. But because the "Find More" link points to page 2, the starting page for this search engine is 1. Double-click your new plugin to show the internal code. Look for the <Start> key and change the <integer> under it to 1. Then save the plugin.
Step 3: Let's test the plugin. Press the Test button at the top of the Plugins & Scanners window. The results show the created plugin is functioning but it's reporting results from some pages we don't need. You can sort the results by location for easier handling.
Next, Control-click one of the entries that begins with https://wiby.me and choose Exclude Domain. This keeps the site's own pages from being reported as matches. You may also see other URLs that could be excluded. Also, if there's a site that may have legitimate pages but you see links to pages that will not match your search, e.g., a contact page, you can select them, Control-click, and Exclude URL. This just ignores those specific pages.
After you've done your exclusions, press the Retry button. You should see a reasonable number of links, often between 10 and 25, being reported for the search URLs in the log. If you need to, you can exclude other URLs or domains and retest. If it doesn't improve things, you can always include them again via the context menu.
Step 4: Now choose File > New Search. Select our new plugin from the options under the magnifying glass. Run a test search using the search term awk. We are now seeing plenty of results and you have just created a search plugin from scratch.
Not every search engine is so simple but there are other keys you can use to fine-tune results. This will require digging more deeply into the underlying code of the web page showing your search results.
You can select Web > Source to show the HTML source and search for items in the content of the page. But if you want to use some very powerful tools, enable Developer Extras in Settings > Web, so you can Control- or right-click and choose Inspect Element on parts of the rendered page in a DEVONagent Pro browser window. The Web inspector tools will open showing the page source and focus on the inspected item. As you select items in the markup in the developer tools, the item on the web page will be highlighted. You can also select the crosshair button and click on an item on the web page to focus it.
Now let's look at a pair of DEVONagent Pro keys you can use to isolate the search results from other parts of a web page.
Link Boundaries: When you examine the markup of the search results, look for HTML elements bookending the list of results. This ignores possible matches in headers, footers, sidebars, etc. You will specify these asLinksStart and LinksEnd keys.
Imagine you examine a specific page and notice it begins with a section about the search results starting with <p class="pin">. Scrolling down you find an element after the results section starting with <a class="more". We can use those elements to focus our searching.
Double-click your plugin and add these keys and values. Since you're editing an XML-based plugin, you need to properly enter the greater than and less than signs as HTML elements. Also, "globbing" isn't supported in all keys but you can see with LinksEnd, a partial match is sufficient as it isn't closed with >.
After you finish your edits, save the plugin. If you've made an error, it will display at the bottom of the editor. If entered correct, give the search another try and check for better results.
As mentioned above, you can examine the HTML source code of results and try to find strings that can be used for these keys. A few other useful keys used to more accurately target elements or properties are TitleStart and TitleEnd to extract the title of the article, DateStart and DateEnd if the date is found in the document's content.
When looking at the topics in the digest pane after you've done your searches, you can set the NoTopics list. Any word that should not be included as a topic can be added to this list. Select words to exclude in the Digest tab of a search window and press ⌘︎C to copy them. You can then paste them into a <string>, as one comma-delimited line, like so…
In other cases, you might also want to look at the source of your plugin and validate the values for Start, OffsetPerPage, and ResultsPerPage, etc. Adding a CrawlDelay key to slow the pace of contact with the search engine's servers. This can help avoid overloading it and make DEVONagent Pro appear to be a bit more like a human is searching.
In addition, DEVONagent Pro may not always be able to insert the _agentOffset_ and _agentNumber_ placeholders automatically. You might have to do this on your own; otherwise, only the first results page is used.
Finally, uncheck any operators the search engine doesn't support.